Advanced artificial intelligence models score well on professional medical exams but still flunk one of the most crucial physician tasks: talking with patients to gather relevant medical information and deliver an accurate diagnosis. "While large language models show impressive results on multiple-choice tests, their accuracy drops significantly in dynamic conversations," says Pranav Rajpurkar at Harvard University. That became evident when researchers developed a method for evaluating a clinical AI model's reasoning capabilities based on simulated doctor-patient conversations. The "patients" were based on 2000 medical cases primarily drawn from professional US medical board exams. "Simulating patient interactions enables the evaluation of medical history-taking skills, a critical component of clinical practice that cannot be assessed using case vignettes," says Shreya Johri, also at Harvard University.

As medicine continues to test automated machine learning tools, many hope that low-cost support tools will help narrow care gaps in countries with constrained resources. But new research suggests it's those countries that are least represented in the data being used to design and test most clinical AI -- potentially making those gaps even wider. Researchers have shown that AI tools often fail to perform when used in real-world hospitals. It's the problem of transferability: An algorithm trained on one patient population with a particular set of characteristics won't necessarily work well on another. Those failures have motivated a growing call for clinical AI to be both trained and validated on diverse patient data, with representation across spectrums of sex, age, race, ethnicity, and more.

You're reading the web edition of STAT Health Tech, our guide to how tech is transforming the life sciences. Sign up to get this newsletter delivered in your inbox every Tuesday and Thursday. A growing patchwork of state-level privacy laws could pose a challenge for upstart health tech companies. After California led the way with its consumer privacy law in 2018, Virginia and Colorado followed suit, and now Massachusetts has advanced its own data privacy bill. Each independent piece of legislation could impact consumer-oriented health apps that don't fall under HIPAA -- leading digital health companies to worry about mounting costs to navigate the regulatory thicket and declining revenue for resale of consumer data, Mohana reports.

There's been plenty of hype surrounding the promise of AI in healthcare. That's according to a group of doctors and data scientists who recently published a commentary in BMJ Health & Care Informatics calling on hospitals to establish "organized and dedicated" clinical departments whose primary focus is the successful implementation of AI. "Within academic medicine," they argue, "algorithms are currently developed in silos by researchers interested in the intersection of healthcare and machine learning. This has led to a panoply of published models trained on health data, yet only a handful have been prospectively evaluated on patients.'' In their view, the general "lack of clinical results is the byproduct of a lack of coherence, leadership and vision. Hence, unless we change course, we should expect that AI deployment in healthcare will progress much the way the EHR revolution did before it, that is, mainly based on corporate and administrative benefits without requiring any demonstrable improvements in processes or outcomes for our patients or ourselves." Given the range of challenges facing the implementation of AI in healthcare, the group says any department of AI's primary focus should be on making healthcare organizations "AI Ready." Specifically, resulting "initiatives should lead to the development of models that will directly benefit the health of our patients, pioneer research that advances the field of clinical AI, focus on its integration into clinical workflows and foster educational programs and fellowships to ensure we are training current practitioners as well as the next generation of leaders in this field." Moreover, dedicated departments should be front and center when it comes to the implementation, utilization and enhancement of the infrastructures that underlie AI solutions. "Central to this mission will be removing barriers to data access, and the proposed department would therefore assume partnered stewardship of the institution's data as part of its mandate.

A group of doctors and data scientists is calling on hospitals to create clinical departments devoted to artificial intelligence to harness the power of the technology to transform patient care. While there have been many predictions of AI's potential to benefit health care delivery -- from helping doctors perform surgery to catching cancer earlier -- the technology's benefits so far have been blunted by inconsistent implementation, the researchers say. They outline a plan to make hospitals "AI ready," in a way they say would enhance both patient care and medical research. UVA Health's Dr. David J. Stone and colleagues from several other major medical centers outline their plan in "The Clinical Artificial Intelligence Department: A Prerequisite for Success," published in BMJ Health & Care Informatics. They begin by offering a frank assessment of the current integration of AI in health care: "The reality of the available evidence increasingly leaves little room for optimism," they write.

A group of doctors and data scientists is calling on hospitals to create clinical departments devoted to artificial intelligence (AI) to harness the power of the technology to transform patient care. While there have been many predictions of AI's potential to benefit healthcare delivery – from helping doctors perform surgery to catching cancer earlier – the technology's benefits so far have been blunted by inconsistent implementation, the researchers say. They outline a plan to make hospitals "AI ready," in a way they say would enhance both patient care and medical research. UVA Health's David J. Stone, MD, and colleagues from several other major medical centers have outlined their plan in a new article in the scientific journal BMJ Health & Care Informatics that was highlighted in the July 22 issue of the STAT health news site's Healthtech newsletter. They begin by offering a frank assessment of the current integration of AI in healthcare: "The reality of the available evidence increasingly leaves little room for optimism," they write.

The role of artificial intelligence in healthcare continues to evolve as does the definition of what it is and is not. This state of flux has contributed to slower than desired adoption, unmet expectations, and gaps between deployment and value realization. If clinical artificial intelligence more specifically is to transform patient care, it must deliver insights that are unique, individualized, can be tied to community and align with existing workflows. Join Jvion, a market leader in clinical AI, along with leadership from Microsoft, for a one-hour webinar that will provide clarity and guidance to aid in addressing patient risk and improving population health across communities.